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In the second paragraph of the Preface, (Code MIR-M) should read 
(Code MAR-I), 

Page 1-10: The sentence below item 5.53 should read: 

Suppose sum of Column A = 76, and sum of Column C = 60. 8 
C — A x 100 or 60. 8 — 76 x 100 = 80 percent. 

Page 1-11: In the Column A heading, delete (1-10). 

Under Col umn C, across from 14. 0, 6.0 should be 6.2 
Under Column C, across from Totals, 76.9 should be 76.8 
Under Column D, across from 14. 0, 2. 0 should be 1. 8 

Page 2-3/2-4: In the heading, the word "Reliability" should be "Quality" 

Page 2-5/2-6: In the Column A heading, delete (1-10). 

Page 2-32; Under CoLumn A, across from 7.33, enter the number 8. 


NASA -Langley, 1964 


Issued 2-14-64 



OFFICE OF MANNED SPACE FLIGHT 


QUALITY PROGRAM EVALUATION PROCEDURES 

(R-2) 


PREFACE 


Quality Program Evaluation Procedures and related survey checklists are 
established as a standard to assure consistent evaluations of quality proce- 
dures and controls being applied to Manned Space Flight Programs. More 
specifically, the objectives are threefold: 

a. To establish uniform standards for evaluating the degree and 
effectiveness of quality practices and controls. 

b. To identify quality type problems for evaluation and correction. 

c. To permit e valuation of various- methods of controlling a specific 
quality area, leading to improved reliability and safety levels. 

This standard is based on and is consistent with NASA Publication 
NPC 200-2; however, it may be used to survey contractual compliance 
to all quality publications. It is designed to identify problem and im- 
provement areas consistent with the severe reliability and safety re- 
quirements of manned space flight systems . 

Comments and questions concerning the requirements set forth in this 
publication should be referred to the Office of Manned Space Flight 
(Code MIR-M), NASA Headquarters, Washington 25, D.C. Questions 
concerning its application to specific contracts should be referred to the 
cognizant NASA Center. 
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1. QUALITY PROGRAM EVALUATION PROCEDURES 


1.1 INTRODUCTION 

The reliability and quality requirements of the Manned Space Flight Program demand 
design, manufacture, test, and operations reliability and quality levels that far ex- 
ceed those required in the past. The Quality Program Evaluation Procedures pre- 
sented in this section have been developed to assist in assessing the degree that 
quality and inspection programs are properly related to the over -all needs of the 
Manned Space Flight Program, and to provide a base from which the specific action 
required to make the programs more effective can be developed. 

1 . 2 RESPONSIBILITIES 

1.2.1 OMSF Responsibilities 

The Office of Manned Space Flight will: 

a. Be responsible for the establishment and revision of evaluation 
procedures. 

b. Insure follow-up review of the quality program requirements to determine 
the actions taken on deviations noted during the evaluation. 

c. Monitor scheduling and maintain a record of all evaluations and follow-up 
reviews. 

1.2.2 NASA Center Responsibilities 

The cognizant NASA Centers will be responsible for implementing an effective program 
of periodic quality program evaluations (surveys). More specifically, this will involve 
the following responsibilities: 

a. Schedule surveys. 

b. Designate a chairman and direct the survey team. 

c. Notify the contractor, OMSF, and, if applicable, the cognizant Govern- 
ment representative by letter at least 30 days prior to the date of the 
proposed survey. 
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d. Conduct a presurvey conference at which team members will meet with 
contractor personnel at the contractor* s facility. The survey team 
chairman will explain the objectives and general plan of the evaluation 
for the understanding of all concerned. 

e. Conduct the surveys and evaluate the quality program utilizing the pro- 
cedures and checklists outlined herein. 

f. Conduct a postsurvey critique with contractor personnel and discuss the 
preliminary results of the evaluation. The contractor should be given 
an opportunity to explain any unusual or discrepant information obtained. 

g. The chairman of the survey team will be responsible for preparing a 
final report of the evaluation for the NASA Center and OMSF . Copies 
of this report will be sent to evaluation team members and other activ- 
ities as necessary. The contractor will be notified in writing of the 
results of the evaluation and action necessary to correct or improve 
deficiencies. 

h. Follow-up specific survey results to determine the action taken as a 
result of deficiencies noted during the survey. The assistance of the 
cognizant Government representative and resident Apollo System Project 
Office, if applicable, will normally be utilized to the maximum extent 

in this follow-up. 

i. Maintain records of all survey reports and related follow-up summaries. 

j. Transmit to OMSF copies of all survey reports and related follow-up 
summaries, with copies to other NASA Centers and team members as 
appropriate. 

1.2.3 Survey Representatives 

Survey teams will usually be comprised of the following members: 

a. NASA Center chairman and designated representatives. 

b. Cognizant Government representative (if applicable). 

c. OMSF representatives. 

1.3 AC TIVITY AREAS 

Quality program activities consist essentially of a network of interrelated procedures 
and controls that are designed to assure an end product which meets Manned Space 
Flight Program needs. This quality program extends throughout the entire organiza- 
tion in its Work Element coverage; its time -phased coverage extends from initial 
contract definition throughout the entire program. 
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A quality program can be considered to consist of 15 major Activity Areas, each bear- 
ing a separate and distinct relationship to the over-all program. These Activity Areas 
are described in NPC 200-2 and are listed as follows: 

1 . 0 Introduction 

2.0 Basic Requirements 

3.0 Management 

4.0 Design and Development Control 

5.0 Control of Contractor Procured Material 

6.0 Control of Government Furnished Property (GFP) 

7.0 Control of Contractor-Fabricated Articles 

8.0 Nonconforming Material 

9.0 Inspection, Measuring, and Test Equipment 

10.0 Inspection Stamps 

11.0 Preservation, Packaging, Handling, Storage, and Shipping 
12. 0 Statistical Planning and Analysis 

13.0 Training and Certification of Personnel 

14.0 Data Reporting and Corrective Action 

15.0 Audit of Quality Program Performance 

An evaluation of the Degree of Effective Coverage for each Activity Area can be estab- 
lished by determining the importance of individual Work Elements and the determina- 
tion of the Degree of Effective Coverage provided for each within individual Activity 
Areas. Similarly, an over-all quality program evaluation of an entire quality pro- 
gram can be developed from compiling the results of the individual Activity Area 
evaluation. 

1.4 EVALUATION PROCEDURE 
1.4.1 Objectives 

Quality evaluations are conducted to provide means of: 

a. Determining effective quality program coverage in terms of Manned 
Space Flight Program needs. 

b. Determining effective quality program coverage in terms of specific 
contractual requirements. 

c. Determining the relative strengths and weaknesses in each of the major 
Activity Areas. 
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d. Determining the relative strengths and weaknesses of the individual 
Work Elements which make up each Activity Area. 

e. Measuring, through subsequent surveys, changes in effectiveness of 
quality activities. 

f. Making recommendations for improving, strengthening, or de-empha- 
sizing Aclivity Areas. 

1.4.2 Evaluation Procedure Steps 

In meeting the above objectives, the quality program evaluation is performed in the 
following basic steps: 

a. Determining the Relative Importance (in percent) of each of the quality 
Activity Areas to the specific program in terms of Manned Space Flight 
Program needs. 

b. Determining the Relative Importance (1 to 10) of the individual Work 
Elements vithin each of the Activity Areas. 

c. Establishing the Degree of Effective Coverage (0, 20, 40, 60, 80, or 
100 percent) of the individual Work Elements within each of the Activity 
Areas. 

d. Listing re.ated document number and date (where applicable) for indi- 
vidual Work Elements. 

e. Determining the current assignment of Functional Responsibility for 
each of the individual Work Elements. 

f. Developing a Weighted Effective Rating for each Work Element by multi- 
plying the Relative Importance Factor by the Degree of Effective 
Coverage. 

g. For each Work Element, subtracting the Weighted Effective Coverage 
Rating from the Relative Importance Factor to evaluate each Work 
Element ir terms of need for action and priority. (The higher the num- 
ber, the greater the need. ) 

h. Developing (similar to Step f) a composite Weighted Effective Coverage 
Rating for each Activity Area based on Effective Coverage Ratings on 
individual A'Ork Elements. 

i. For each Activity Area (similar to Step g), subtracting the Weighted 
Effective C overage Rating from the Relative Importance Factor to 
evaluate es.ch Activity Area in terms of need for action and priority. 

(The highe ~ the number, the greater the need.) 


1-4 



j. Developing and establishing specific recommendations to increase the 
effectiveness of the Manned Space Flight Quality Program. 

k. Reviewing survey reports to identify Activity Areas and Work Elements 
where improved reliability or quality procedures and controls are needed. 

l. Reviewing survey reports to determine Activity Areas and Work Ele- 
ments where exceptionally effective quality procedures and controls have 
been identified. 

1-4.3 Contractual Compliance Procedure Steps 

In determining compliance to specific contractual requirements, only a slight modi- 
fication to the above procedure is required and is accomplished as follows: 

a. The individual Work Elements within the Activity Areas are re- 
collated, as applicable, against the requirements of the specific con- 
tractual documents, thereby replacing the Activity Areas by Contractual 
Requirement Areas. 

b. Establishment of Relative Importance Factors for the Work Elements 
within the Contractual Requirement Areas. 

c. Upon completion of Step e, in paragraph 1.4.2, the results of Step c 
of that procedure (Degree of Effective Coverage), Step d (applicable 
document number and date), and Step e (assignment of Functional 
Responsibility) are posted to this revised breakdown. 

d. Steps f through 1 are then repeated on a Contractual Requirement Area 
basis instead of an Activity Area basis. 

Contractual requirement evaluations will supplement, not replace, the procedure of 
paragraph 1.4. 2 because of the greater coverage required by increased reliability 
and safety needs of Mann 3d Space Flight Programs. 

1.5 SCORING METHODS 

1-5-1 Determining the Relative Importance of Activity Areas 

Each of the Activity Areas listed in paragraph 1. 3 has a separate and distinct contri- 
bution in a quality program. However, all of these Activity Areas are not independent 
and a major weakness wilhin a quality program in one of these Activity Areas can 
have a decided effect upon the contribution of the other Activity Areas. 
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In the implementation of the quality program survey, the first step is to establish 
Relative Importance Factors for each Activity Area. For initial planning purposes, 
a set of Relative Importance Factors has been established as shown on page 2-5. 

These Relative Importance Factors can be revised in later surveys to reflect adjust- 
ments indicated for the type of program being surveyed. 

1.5.2 Determining the Relative Importance of the Individual Work Elements 
of Each Activity Area 

Each of the Activity Areas is made up of a number of Work Elements. These Ele- 
ments describe the key quality procedures and controls that are necessary to obtain 
maximum results from the Activity Area. The Work Elements are in the form of 
numbered declarative statements with an affirmative response scoring favorably. 

The number of Work Elements used for each Activity Area varies with the complexity 
of the Activity Area. The Work Elements are grouped under subheadings for easy 
reference. It is intended that the Work Elements, but not the subheadings, be scored. 
These Work Elements can be used in a survey regardless of the specifications and/ or 
other requirements of the contract, as indicated in paragraph 1.4.3. Work Elements 
within an Activity Area are not equally important. Initial Relative Importance Factors 
have been established for each Work Element as noted in Column A of the Survey 
Checklists (Section 2). These factors will be subject to revision based on survey 
experience. 

1.5.3 Rating Work Elements for Degree of Effective Coverage 

The successful application of a quality program survey in fulfilling its objectives as 
a program status and improvement tool lies in the logical and accurate evaluation of 
the Degree of Effective Coverage currently provided against each Work Element. 

It should be recognized that the assignment of the Relative Importance Factor for 
Work Elements (paragraph 1.5.2) and the determination of their current Degree of 
Effective Coverage are directed at establishing those Work Elements and those 
Activities that merit the highest priority of action to strengthen the program coverage 
and effectiveness. Some Work Elements of an Activity Area may be highly important 
but have a low Degree of Effective Coverage. It is those Work Elements which have 
the hi ghest combination of Relative Importance and lack of Effective Coverage that will 
merit highest priority of action. 
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As shown in Column B of Figure 1-1, when rating the Degree of Effective Coverage, 
each Work Element will be given a rating of 0, 20, 40, 60, 80, or 100 percent. This 
rating expresses, in percentage points, the Degree of Effective Coverage of the Work 
Element found by the survey team personnel. Satisfactory coverage warrants a rating 
of 100 percent. Ratings less than 100 percent should be supported by adequate notes. 

The product of Column B and Column A is now inserted in Column C to give a quantity 
which represents the Weighted Effective Coverage of the Work Element. 

1.5.4 Determining Document Numbers and Dates 

Where applicable to individual work statements, the related document numbers, titles, 
and dates which reflect compliance are recorded as indicated in Figure 1-1. This 
might be a special or periodic report or procedure, for example. 

1.5.5 Determining Current Assignment of Functional Responsibility for 

Work Elements ” ' 

The Column E (Function Responsible for Work Element) will be filled in to indicate 
the organization unit responsible for the Work Element. 

The possibility of multiple assignments or the lack of assignments must also be re- 
corded, as appropriate. Important details that cannot be stated in this column should 
be supplied in a supporting narrative report which will be referenced in Column E. 

1.5.6 Determining the Relative Need for Action of the Individual Work Elements 
of an Activity 

When reviewing the results of surveys or when recommending corrective action, it is 
desirable to point out the extent of lack of coverage of Work Elements and to include at 
the same time the Relative Importance aspect of the Work Elements. To do this, sub- 
tract Weighted Degree of E ffective Coverage (Column C) from the Established Import- 
ance Factor (Column A). The result is then placed in Column D. The higher the 
number, the greater the ne ed. 

1.5.7 Developing Activity Area Effective Coverage Ratings 

The Degree of Effective Coverage of an Activity Area is a function of the Degree of 
Effective Coverage of its individual Work Elements, weighted by their Importance 
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Factor. The mechanics of determining the Activity Area Degree of Effective Coverage 
are shown in Figure 1-2 and are detailed as follows: 

a. Add the Weighted Degree of Effective Coverage for all Work Elements 
(Column C) ;o obtain a total for the Activity Area. 

b. Add the Relative Importance Factors for all Work Elements (Column A) 
to obtain a total for the Activity Area. 

c. Divide the Activity Area total for Weighted Degree of Effective Coverage 
(step a above) by the Activity Area total for the Relative Importance 
Factor (step b above). The result is the Activity Area Degree of Effec- 
tive Coverage and should be noted on the last page of the Activity Area 
Work Elemeit sheets. As shown on Figure 1-3, the Activity Area 
Degree of Etfective Coverage number should also be inserted in Col- 
umn B of the quality program evaluation summary sheet. 

1-5.8 Rating Activity Areas in Terms of Need for Action 

The procedure for rating Activity Areas in terms of need for action is identical to the 
procedure for rating the Work Elements. The Relative Need (Column D) is the differ- 
ence between the Established Importance Factor and the Weighted Degree of Effective 
Coverage (Column A - Column C). 

1-5.9 Determining Over-all Degree of Effective Coverage 

The over-all Degree of El ective Coverage for each quality program is established 
directly from the Relative Importance Factors of the individual Activity Areas and 
their associated degree of current Effective Coverage. It is developed in the same 
manner as the Degree of Effective Coverage of the Activity Areas was established, 
except all Importance Faclors must add to 100 percent as shown in Figure 1-3. 

The Relative Importance Factor (Column A) established for each Activity Area is 
multiplied by the Degree o ' Effective Coverage determined for that Activity (Column B) 
and the result, representing a Weighted Effective Coverage figure in percent, is in- 
serted in Column C . 

The sum of Weighted Effective Coverages (Column C) is then totaled and is a percent- 
age representing the degree of over-all quality program coverage. 
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2. QUALITY PROGRAM EVALUATION CHECKLISTS 


This section contains the Quality Program Evaluation Summary Sheet for rating and 
evaluating the fifteen (15) Activity Areas. It also contains the individual Activity Area 
checklists for use in rating and evaluating the Work Elements that comprise each area. 
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criteria, acceptance limits, specific test equipment requirements, and identification 
of the article. 
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4. 13 Action is taken to maximize the use of standard or approved materials, parts and 
processes. 

4. 14 Applicable Government specifications are reviewed, and necessary supplements and 
interpretations are provided to define the quality requirements of the purchased or 
fabricated articles. 
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, packaging, and transportation of articles. 



QUALITY PROGRAM EVALUATION 





QUALITY PROGRAM EVALUATION 



design function and to the supplier. 
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during all phases of fabrication. 
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. 28 An End Item Test Plan provides a technical description of the system, assemblies and 
subassemblies and is submitted to customer for approval prior to start of tests on 
completed end items. 
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7. 37 Any repairs, retests or modification after the final tests may require a retest or 
reinspection at the discretion of the authorized NASA representative. 
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12. 8 Sampling methods used are derived from sampling programs which have proven to be 
statistically sound and afford reliability maintenance of acceptable levels. 
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